Skip to content

Stop the asset page's KPIs counting a day the chart does not show - #2434

Merged
Flix6x merged 14 commits into
mainfrom
fix/kpi-window-off-by-one
Sep 1, 2026
Merged

Flix6x merged 14 commits into
mainfrom
fix/kpi-window-off-by-one

Conversation

@Flix6x

@Flix6x Flix6x commented Aug 23, 2026

Copy link
Copy Markdown
Member

Description

Closes #2453.

KPIs on the asset page cover one day more than the chart beside them, and, as found in review, do not total what the chart draws at all when a sensor has more than one source or a revised belief.

  • ui/graphs: pass the chart's end date to the KPI endpoint unchanged
  • api/v3_0: total the beliefs the chart draws, rather than belief rows grouped by data source
  • Added changelog item in documentation/changelog.rst

getAssetKPIs advanced its end date by a day, to "make the end date inclusive". But both callers already hand it the chart's exclusive end — the picker applies +1 day before calling, and event_ends_before is exclusive on the initial page load. The endpoint ends its own window before end as well: _get_sensor_stats filters TimedBelief.event_start < end_dt.

Advancing the date also mutated it in place, and that Date object is the one held by storeEndDate and previousResult.end, so the chart's own end date moved a day forward on every selection. Not advancing it at all fixes both, which is why this is a one-line change.

Closes #2453

Where the extra day came from

The advance was not arbitrary, and it was once correct. Until #1909 this function took no arguments and read the picker directly:

var endDate = picker.getEndDate();
endDate.setDate(endDate.getDate() + 1);

picker.getEndDate() is the last day selected — 7 January for a 1–7 January selection — so advancing it produced the exclusive end the endpoint wants.

#1909 ("fix: pass start and end dates to getAssetKPIs() on graphs page", closing #1908, shipped in v0.30.3, January 2026) gave the function startDate/endDate parameters, because reading the picker broke page load. Both callers pass the chart's already-exclusive end: the picker handler applies +1 day before calling, and event_ends_before is exclusive on the initial load. The advance inside was carried over unchanged, and from that point added a day too many.

So this is a regression from #1909 rather than the undoing of a deliberate decision. Its own changelog entry stays as it is: #1909 shipped in v0.30.3, so this is a fix to released behaviour, not to an unreleased regression. I have left a comment where the advance used to be, so it does not get put back.

Warning

This changes reported KPI values. They were too high (or too wide, for min/max) by one day's worth of data. Anything tracking them will step down when this lands. The old numbers were wrong, but the change is visible.

Look & Feel

On a seeded asset carrying one unit of energy per day, asking for a three-day window:

end sent KPI total
T0+3d (what the chart shows) 3.0
T0+4d (what the UI sent) 4.0

How to test

test_kpi_window_end_is_exclusive in flexmeasures/api/v3_0/tests/test_assets_api.py asserts that a three-day window over one daily value per day totals three, not four.

To be explicit about what that test does and does not cover: it pins the endpoint's window semantics, which the page depends on. It does not exercise the JavaScript change, and would pass without it — there is no JavaScript test runner in this repo. The front-end half was verified by hand, by reading the end parameter off the request the page issues.

Manually: open an asset page with KPIs configured, select a range, and compare a KPI against the chart beside it. Before this change a sum covers one day more than the chart draws.

Notes

Split out of #2433 at the maintainer's request, because it changes user-visible numbers while that PR is a performance change. #2433 also carries this fix, so the two converge: graphs.html merges cleanly in either order (verified by a trial merge). The changelog entry lives here, and has been removed from #2433 to avoid a duplicate line.

Flix6x added 2 commits August 23, 2026 18:19
Context:
- getAssetKPIs advanced its end date by a day to "make the end date
  inclusive", but both callers already hand it the chart's exclusive end,
  and the endpoint ends its own window before `end` as well
  (_get_sensor_stats filters event_start < end).
- Every KPI therefore covered one day more than the chart beside it. On a
  seeded asset with one unit of energy per day, a three-day selection
  reported a total of four.
- Advancing the date also mutated it in place, and that Date object is the
  one held by storeEndDate and previousResult.end, so the chart's own end
  date moved a day forward on every selection. Not advancing it at all
  fixes both.

Change:
- Pass the chart's end date through unchanged.
- Assert the endpoint's window semantics, which the page depends on.

Signed-off-by: F.N. Claessen <felix@seita.nl>
Context:
- The entry was added before the PR existed, with an XXXX placeholder.

Change:
- Point it at PR #2434.

Signed-off-by: F.N. Claessen <felix@seita.nl>
Flix6x added a commit that referenced this pull request Aug 23, 2026
Context:
- The KPI day-counting fix was split out into PR #2434, since it changes
  user-visible numbers while this PR is a performance change. Both
  branches carry the same code change and merge cleanly either way, but
  the changelog would have gained the line twice.

Change:
- Drop the entry here. The date-corruption entry stays, as that fix is
  needed by the cache in this PR.

Signed-off-by: F.N. Claessen <felix@seita.nl>
@Flix6x
Flix6x requested a lite review from Copilot August 23, 2026 16:22

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot encountered an error and was unable to review this pull request. You can try again by re-requesting a review.

Note

This error may be related to your runner configuration. You can now configure runners for Copilot code review separately from Copilot cloud agent by creating a copilot-code-review.yml file with your setup steps. Read the docs for details.

@read-the-docs-community

read-the-docs-community Bot commented Aug 23, 2026

Copy link
Copy Markdown

Context:
- The advance was not arbitrary. Until PR #1909 this function read
  picker.getEndDate(), which is the last day selected, so advancing it by
  a day correctly produced an exclusive end.
- PR #1909 (v0.30.3, January 2026) gave the function start and end
  parameters and had the callers pass the chart's already-exclusive end,
  but kept the advance, which from then on added a day too many.

Change:
- Note that history where the advance used to be, so it does not get put
  back.

Signed-off-by: F.N. Claessen <felix@seita.nl>

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot encountered an error and was unable to review this pull request. You can try again by re-requesting a review.

Note

This error may be related to your runner configuration. You can now configure runners for Copilot code review separately from Copilot cloud agent by creating a copilot-code-review.yml file with your setup steps. Read the docs for details.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot encountered an error and was unable to review this pull request. You can try again by re-requesting a review.

Note

This error may be related to your runner configuration. You can now configure runners for Copilot code review separately from Copilot cloud agent by creating a copilot-code-review.yml file with your setup steps. Read the docs for details.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot encountered an error and was unable to review this pull request. You can try again by re-requesting a review.

Note

This error may be related to your runner configuration. You can now configure runners for Copilot code review separately from Copilot cloud agent by creating a copilot-code-review.yml file with your setup steps. Read the docs for details.

@Flix6x

Flix6x commented Aug 23, 2026

Copy link
Copy Markdown
Member Author

Copilot review requested three times (17:22, 17:31, 17:35 UTC); each attempt came back with "Copilot encountered an error and was unable to review this pull request", referring to runner configuration. Requests on #2433 and #2435 in the same window succeeded, including ones touching this same file, so it looks specific to this PR or transient on their side rather than something in the diff.

Not retrying further to avoid noise. @Flix6x, worth a re-request from your side later, or a human review — the change is one line plus a test.

CI is green: 13/13 checks pass.

Context:
- toIsoStringWithOffset appended the local UTC offset to date.toISOString(),
  which is UTC, without moving the clock time, so the string named an
  instant wrong by exactly that offset.
- getAssetKPIs is its only caller, so this shifted the very window this PR
  is about. West of UTC the shift lands on a different day for sensors of
  daily resolution, so KPIs could report a different day than the chart.

Change:
- Write the local clock time, then append the offset.
- The same fix is in PR #2435, where the JavaScript tests that found it
  live. Both branches carry identical text, so they merge either way. The
  changelog entry for it stays in #2435.

Signed-off-by: F.N. Claessen <felix@seita.nl>
@Flix6x

Flix6x commented Aug 23, 2026

Copy link
Copy Markdown
Member Author

Added the toIsoStringWithOffset fix here too, per @Flix6x. It is a second, independent error in the same KPI window: this PR fixes an end date a day too late, that one fixes both ends being shifted by the viewer's UTC offset. West of UTC the shift lands on a different day for daily-resolution sensors, so the two compound.

The file is byte-identical to the version in #2435, where the JavaScript tests that found it live, so the branches merge in either order. The changelog entry for it stays in #2435, to avoid the line appearing twice.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot encountered an error and was unable to review this pull request. You can try again by re-requesting a review.

Note

This error may be related to your runner configuration. You can now configure runners for Copilot code review separately from Copilot cloud agent by creating a copilot-code-review.yml file with your setup steps. Read the docs for details.

@Flix6x

Flix6x commented Aug 23, 2026

Copy link
Copy Markdown
Member Author

Update on the Copilot review: five attempts now, across three different commits, spanning about two hours — every one returned "Copilot encountered an error and was unable to review this pull request".

attempt commit result
16:23 e07e5ad8c errored
17:23 4a242f557 errored
17:30 4a242f557 errored
17:35 4a242f557 errored
18:11 a48553b40 errored

Requests on #2433 and #2435 succeeded during the same windows, including ones touching this same file, so it is specific to this PR rather than a general outage or anything in the diff. I have stopped retrying.

This PR therefore has had no automated review at all, which is worth knowing given it is the one that changes reported KPI values. @Flix6x, a re-request from your account may fare better; otherwise it is worth a human read. The change is two one-line fixes plus a test.

CI is green: 13/13.

Flix6x added a commit that referenced this pull request Aug 23, 2026
Context:
- Review feedback: the entry linked both the PR and the issue it closes.

Change:
- Drop the issue link. The other entries on this branch, and those on
  PR #2434 and PR #2435, already reference only their PR.

Signed-off-by: F.N. Claessen <felix@seita.nl>
@Flix6x
Flix6x requested a lite review from Copilot August 23, 2026 21:11
@Flix6x

Flix6x commented Aug 23, 2026

Copy link
Copy Markdown
Member Author

Copilot could not review this one (five attempts, all erroring), so I ran three adversarial reviewers locally instead — one on the KPI-window claim, one on timezone correctness, one on test quality. Between them they found five things worth acting on. All are now addressed; the core claim survived.

What survived. The claim that both callers pass an exclusive end was checked by rendering the page for FLEXMEASURES_MODE both normal and play and parsing the emitted script: exactly one definition and two call sites in both, both passing an exclusive end. The #1909 history in the comment checks out against git show aa7f8c90e. And toIsoStringWithOffset survived 1,151,931 assertions across 14 timezones — every DST transition instant in 2019–2027 ±3h, sub-hour offsets (+05:45, +13:45), the repeated hour, and padding edges — with zero failures, and its output verified to parse to the right instant through the real AwareDateTimeField.

What changed.

  1. The test proved nothing about the fix. Both fixes are JavaScript; the test drove the endpoint, which this branch does not touch. It passed on main and would pass with both fixes reverted. The repo asks that a new test be shown to fail first. There is now a rendered-page assertion that getAssetKPIs passes its window on untouched — reintroducing endDate.setDate(endDate.getDate() + 1) turns it red, which is exactly how the regression fix: pass start and end dates to getAssetKPIs() on graphs page #1909 introduced would have been caught. No new dependency; it reads the page the Flask test client renders.

  2. The endpoint test could not see a shifted window. Every belief was worth 1.0, so it counted days rather than identifying them — any three-day window passed. Days now carry distinct values, so the right window totals 6.0 and an hour-shifted one totals 9.0, with an explicit total != shifted assertion so that discrimination cannot rot. Proved red by inverting the endpoint's event_start < end_dt to <=.

  3. Three comments described behaviour that no longer exists. Two still gave "getAssetKPIs bumps storeEndDate by +1 day" as the reason fastChartWindow is snapshotted as epoch numbers — load-bearing, since a maintainer trusting it could drop the snapshot or restore the advance. One is on this branch, the other was on Load only the newly selected days when a chart's time range changes #2433, fixed there in 9f3ecd2 with matching wording so the two stay identical.

  4. The comment overstated the agreement. It said the endpoint ends its window "exactly as the chart does". The predicates differ — the chart keeps events whose end falls on the window's end, the endpoint keeps events whose start falls before it — and they agree only for the daily, midnight-aligned sensors this endpoint documents. For an instantaneous sensor the KPI is now one event short of the chart. Reworded to say so.

  5. Fixture borrowing. The new rendering test took an asset from a shared fixture that earlier tests in the same module delete; it built its own. Both files pass under random ordering.

One thing I am leaving, and flagging instead. Removing the advance also stops storeEndDate being corrupted for the rest of the page session, which means the chart's own reloads after an upload or delete, the chart-type re-embed, and the replay bounds were all a day too wide on KPI-enabled asset pages. That is real and now fixed, but its changelog line lives on #2433 ("the chart's selected end date was pushed a day forward on every date selection"), so if this PR merges alone that symptom ships undocumented. It also makes the previousResult.end.getTime() === endDate.getTime() short-circuit in fetchGraphDataAndKPIs reachable on KPI pages for the first time, since that comparison could never succeed while the date was being mutated.

99 tests pass across the two touched modules, under random ordering.

@Flix6x
Flix6x requested a review from nhoening August 24, 2026 07:49
Flix6x added a commit that referenced this pull request Aug 31, 2026
Context:
- Review by @nhoening: PR #2434 now isolates the KPI window fix, so this PR
  should carry neither the changelog entry nor the code, nor its test.
- Almost right. What this PR genuinely needs is narrower than the fix: the
  advance must not reach storeEndDate or previousResult.end, because both
  are handed to the cache, which would then record a span a day wider than
  it holds. It does not need the advance gone.

Change:
- Restore getAssetKPIs to what main has, advance and all, so the fix and
  its user-visible effect belong wholly to PR #2434.
- Pass copies at both call sites, which changes no KPI value and keeps the
  chart's own dates out of reach. Once #2434 lands these are belt and
  braces, and the two branches merge either way.
- Drop the changelog entry and test_kpi_window_end_is_exclusive, which
  #2434 carries.
- Say why replay fetches around the cache rather than through it.

Signed-off-by: F.N. Claessen <felix@seita.nl>
Flix6x added a commit that referenced this pull request Sep 1, 2026
…2433)

* ui/graphs: stop the KPI fetch from corrupting the chart's date range

Context:
- getAssetKPIs() advances the end date it is given by one day, to make the
  KPI window inclusive.
- On the asset page, it was handed the very Date objects that storeEndDate
  and previousResult.end hold, so after one date selection both were a day
  late. The exact-range short-circuit in fetchGraphDataAndKPIs then stopped
  matching, and embedAndLoad sliced a day too much off the cached data.
- The initial call additionally read storeStartDate/storeEndDate from the
  module body, which is evaluated before the DOMContentLoaded handler that
  assigns them (verified in headless Chrome), so on an asset with KPIs it
  threw on undefined and aborted the rest of the module.

Change:
- Pass copies to getAssetKPIs at both call sites.
- Derive the initial window from the template values rather than from the
  not-yet-assigned globals.

Signed-off-by: F.N. Claessen <felix@seita.nl>

* ui/graphs: fetch chart data through a single module

Context:
- Four call sites in graphs.html (initial load, sensors-to-show reload,
  replay, and date selection) each built the same /chart_data query string
  by hand and repeated the same response handling.
- That duplication is what makes a reuse cache awkward to add: it would
  have to be threaded through every call site separately.

Change:
- Add chart-data-source.js with buildChartDataUrl, fetchChartData and
  fetchChartAnnotations, and route all five fetches through it.
- Drop the now-unused decompressChartData import, the dead module-level
  queryStartDate/queryEndDate, and two parameters of fetchGraphDataAndKPIs
  that the caller no longer needs.
- No behaviour change: the generated URLs are unchanged.

Signed-off-by: F.N. Claessen <felix@seita.nl>

* ui/graphs: fetch only the newly selected days, not the whole window

Context:
- Issue #101: changing the selected time window re-queried the entire
  window, even when almost all of it was already loaded. Only an exactly
  identical selection was served from memory.
- Profiling a 5-sensor asset holding a year of 15-minute data: a request
  costs ~105 ms regardless of size, plus ~24 us per record. Re-fetching a
  year takes ~4.4 s and 14 MiB, of which ~93% is server time.

Change:
- Add chart-data-cache.js, which works out what a newly selected window
  adds to the loaded one and fetches only that.
- The date picker only yields whole nominal days, so the loaded window is
  always one contiguous range; interval arithmetic stays on Date objects
  so a 23- or 25-hour day across a DST transition stays correct.
- Merged records are de-duplicated, since an event straddling the seam is
  returned by both halves.
- A window fully inside the loaded one now needs no request at all.

Signed-off-by: F.N. Claessen <felix@seita.nl>

* data/tests: pin down that a chart data window can be fetched in parts

Context:
- The UI now fetches only the part of a newly selected window that it does
  not already hold, which assumes that [a, b) plus [b, c) equals [a, c).

Change:
- Assert that splitting a window loses no events and changes no values,
  for resolutions that divide a day and for one (7 minutes) that does not.
- Assert that the event straddling the seam is repeated rather than
  dropped, and that both halves agree on its value, which is what makes
  de-duplicating in the front end sufficient.

Signed-off-by: F.N. Claessen <felix@seita.nl>

* docs: changelog entries for the chart data reuse and the KPI date fix

Context:
- Both changes are user-visible: charts respond faster to a new date
  selection, and the KPI-related date corruption is a bugfix.

Change:
- Add one entry under New features and one under Bugfixes for v1.1.0.
- PR numbers are placeholders (XXXX) until the PR exists.

Signed-off-by: F.N. Claessen <felix@seita.nl>

* ui/graphs: keep the widest span loaded, so zooming back out is free

Context:
- The reuse introduced in the previous commit trimmed its records to the
  window on display, so narrowing the selection discarded everything
  outside it. Zooming into 7 days of a loaded 31-day window and back out
  then re-fetched the other 24 days in two requests.

Change:
- Hold the widest contiguous span fetched so far and clip only what is
  handed to the chart, so a narrowed selection and a return to the earlier
  span both cost nothing.
- The cache is bounded by the widest contiguous span browsed, which is
  what selecting that span in one go would have loaded anyway. Selecting a
  window that does not touch the cached span replaces it, so the cached
  span stays contiguous.
- Reset the cache in reloadChartData, where the sensors shown or the data
  itself may have changed and nothing held is still trustworthy.

Signed-off-by: F.N. Claessen <felix@seita.nl>

* docs: fill in the PR number in the changelog entries

Context:
- The entries were added before the PR existed, with XXXX placeholders.

Change:
- Point both entries at PR #2433.

Signed-off-by: F.N. Claessen <felix@seita.nl>

* ui/graphs: rename chart-data-source.js to chart-data-fetch.js

Context:
- Review feedback: "source" is reserved for belief sourcing in this repo,
  so naming a module after it is misleading.

Change:
- Rename the module and update its two importers.

Signed-off-by: F.N. Claessen <felix@seita.nl>

* ui/graphs: reuse only what a direct fetch would have returned

Context:
- Review feedback: clipping cached records by event_start does not match
  the API, which selects events *overlapping* the window
  (event_ends_after / event_starts_before). A 50-minute sensor lost its
  leading event whenever the window was narrowed, since that event starts
  before midnight and runs into the day.
- Investigating that surfaced a second, larger mismatch: the API anchors
  its resampling at the start of the window asked for, so a coarser sensor
  shown next to a 7-minute one comes back on shifted timestamps for a
  window offset by a fraction of the resolution. Those records cannot be
  reconciled with the ones already held.

Change:
- Clip on overlap rather than on event_start, and match instantaneous
  events inclusively on both edges, as the API does.
- Derive the resolution the events are actually spaced on: every sensor
  with a resolution is resampled to the finest one requested, while the
  response still reports each sensor's own.
- Refuse to reuse records for a window that is not on the same resampling
  grid, falling back to fetching the window whole. Whole-day selections of
  sensors whose resolution divides a day are always aligned, so this only
  bites the cases that were wrong before.

Signed-off-by: F.N. Claessen <felix@seita.nl>

* ui/graphs: seed the cache with the initially loaded window

Context:
- Review feedback: the cache was only ever filled by a date selection, so
  the first selection after a page load found it empty and re-fetched the
  whole newly selected range, even though the initial window was already
  in memory. The same applied after a reload triggered by a data change.

Change:
- Route the initial load and the post-reset reload through the cache, so
  the optimisation applies from the first date change onwards. Neither
  changes what is requested: an empty cache fetches the whole window.

Signed-off-by: F.N. Claessen <felix@seita.nl>

* data/tests: pin down the window semantics the chart cache relies on

Context:
- The front end reproduces a fetch from records it already holds, so it
  depends on three behaviours that were not covered.

Change:
- Assert that a window selects events overlapping it, not only events
  starting inside it.
- Assert that instantaneous events are included on both window edges.
- Assert that resampling is anchored at the requested window start, which
  is why the front end refuses to reuse records across such a shift.
- Reflow two docstrings to break only after punctuation, per the repo
  convention.

Signed-off-by: F.N. Claessen <felix@seita.nl>

* ui/graphs: treat touching windows as one contiguous span

Context:
- Review feedback: reuse required a strict overlap, so a selection stepped
  on by exactly its own width (start === cached.end) counted as disjoint
  and replaced the cache. Stepping back then re-fetched a window that had
  just been thrown away, which is the opposite of holding the widest span.

Change:
- Allow the boundaries to touch, so such a selection extends the cached
  span instead of replacing it. What gets fetched is unchanged: the new
  window is still entirely missing and is still fetched whole.
- Note in missingRanges that the half-open convention has one exception,
  instantaneous sensors, which the API matches inclusively on both edges.

Signed-off-by: F.N. Claessen <felix@seita.nl>

* ui/graphs: stop getAssetKPIs from mutating the window it is given

Context:
- Review feedback: passing defensive copies at both call sites left the
  trap in place, since the next caller has to know to do the same. That is
  what caused the date corruption fixed in 04d6b1b.

Change:
- Advance a copy of the end date inside getAssetKPIs, and drop the copies
  at the call sites, which can now pass the chart's own window dates.

Signed-off-by: F.N. Claessen <felix@seita.nl>

* ui/graphs: break the JavaScript comments only after punctuation

Context:
- The repo's line-break convention was written for Python, but it exists
  for stable review comments and text search, which applies just as much
  to the JavaScript comments added here.

Change:
- Reflow the JSDoc blocks and inline comments in chart-data-cache.js,
  chart-data-fetch.js and the graphs.html module, so that every physical
  line ends at a comma, semicolon, colon or period.

Signed-off-by: F.N. Claessen <felix@seita.nl>

* agents/docstrings: extend the line-break convention to JavaScript

Context:
- The convention only claimed to apply to Python, so JavaScript comments
  were outside it, even though the reasons for it (stable review comments
  and text search) do not depend on the language.

Change:
- Widen applyTo to .js and .html, retitle to "Docstrings and comments",
  and scope the RST/Click/doctest guidance as Python-specific.
- Add a JavaScript section covering JSDoc blocks and // comments, in .js
  files and in the script blocks of Jinja templates, with an example, and
  note that @param/@returns lines stay on one line however long.
- State that automated reviewers should report JavaScript comments that
  wrap mid-phrase, as they already do for Python docstrings.

Signed-off-by: F.N. Claessen <felix@seita.nl>

* agents/docstrings: match how the other instruction files scope themselves

Context:
- Review feedback: the comma-separated applyTo glob was unlike any other
  instruction file here, and a consumer reading it as one literal pattern
  would match nothing, silently disabling the instructions altogether.

Change:
- Use applyTo "**", as the other cross-language instruction files do. The
  headings already separate the Python-specific guidance from the
  JavaScript section, so the scoping does not need to live in the glob.

Signed-off-by: F.N. Claessen <felix@seita.nl>

* ui/graphs: require the options that the fetch helpers cannot work without

Context:
- Review feedback: the helpers defaulted their options object to {} and
  then dereferenced start and end regardless, so the default only served
  to postpone the failure and suggest the argument was optional.

Change:
- Drop the defaults, since every one of these helpers needs a window.

Signed-off-by: F.N. Claessen <felix@seita.nl>

* ui/graphs: stop the KPIs counting a day the chart does not show

Context:
- Review feedback: getAssetKPIs advanced its end date by a day to "make
  the end date inclusive", but both callers already hand it the chart's
  exclusive end, and the endpoint ends its own window before `end` too
  (_get_sensor_stats filters event_start < end).
- Every KPI therefore covered one day more than the chart beside it. On a
  seeded asset with one unit of energy per day, a three-day selection
  reported a total of four.

Change:
- Pass the chart's end date through unchanged.

Signed-off-by: F.N. Claessen <felix@seita.nl>

* api/v3_0/tests: pin down that the KPI window's end is exclusive

Context:
- The asset page derives the KPI window from the chart's, so the two have
  to agree on whether the end is part of the window. Nothing said so.

Change:
- Assert that a three-day window over one daily value per day totals
  three, not four.

Signed-off-by: F.N. Claessen <felix@seita.nl>

* docs: changelog entry for the KPI window fix

Context:
- KPI figures change for anyone using them, so this needs saying.

Change:
- Add a Bugfixes entry for v1.1.0.

Signed-off-by: F.N. Claessen <felix@seita.nl>

* ui/graphs: correct why the initial window is derived from the template

Context:
- Review feedback: the comment claimed this code runs in the module body,
  before the DOMContentLoaded handler assigns storeStartDate/storeEndDate.
  That is wrong. Checked against the parsed page: on the asset page the
  block sits inside that handler, so the globals are already assigned.
- The real reason not to read them is worse. The handler's closing brace
  sits inside a Jinja conditional, the sensor upload block, so the handler
  ends at that point only when it renders. On every other page it runs on
  to the end of the module. Whether this code is inside the handler
  therefore depends on the page, and deriving the window from the template
  values is what works either way.
- Consequently the claim that the chart failed to load on an asset page's
  first visit was wrong too: the globals were assigned by then. The date
  corruption that 04d6b1b also fixed was real and is unaffected.

Change:
- Replace the comment with the actual reason.
- Drop the incorrect half of the changelog entry.
- Tie the KPI window test to the requesting user's account, rather than to
  whichever account the database returns first.

Signed-off-by: F.N. Claessen <felix@seita.nl>

* ui/graphs: read the initial window from the globals already assigned

Context:
- Twice now I claimed this block was not inside the DOMContentLoaded
  handler: first that it ran in the module body, then that the handler's
  extent varied per page because its closing brace sat in a Jinja
  conditional. Both were inferred from indentation and both were wrong.
- Parsing the rendered pages settles it. The handler spans the same region
  on the asset page and on the sensor page, closing unconditionally at the
  end of the timezone check, and the initial window is inside it and after
  the assignment of storeStartDate/storeEndDate in every case.
- The brace that misled me closes the sensor upload click listener. It is
  written at four spaces where the listener is opened at eight, so it
  reads as the end of the handler.

Change:
- Use storeStartDate/storeEndDate, as main did, dropping the second copy
  of the window that was only introduced to work around a problem that
  does not exist.
- Indent the upload listener's closing brace to match its opening, so the
  next reader is not misled the same way.

Signed-off-by: F.N. Claessen <felix@seita.nl>

* docs: leave the KPI window entry to its own PR

Context:
- The KPI day-counting fix was split out into PR #2434, since it changes
  user-visible numbers while this PR is a performance change. Both
  branches carry the same code change and merge cleanly either way, but
  the changelog would have gained the line twice.

Change:
- Drop the entry here. The date-corruption entry stays, as that fix is
  needed by the cache in this PR.

Signed-off-by: F.N. Claessen <felix@seita.nl>

* ui/graphs: keep the cache consistent when selections overlap

Context:
- Date selections are not serialised and their requests are not aborted,
  so two can be in flight at once. The merge read what was held *after*
  awaiting, by which time another selection may have replaced it.
- Selecting [1, 7) and then [20, 25), with the second answered first, left
  the cache claiming [1, 25) while holding only the two ends. Selecting
  [10, 12) afterwards was then served from that gap: an empty chart, with
  no request made. The widening selection itself also returned 2 of its 6
  days, since it merged against the unrelated window.

Change:
- Snapshot what is held on entry and merge against that snapshot, so a
  result's span and its records always describe the same thing.
- Return the window this call assembled, rather than whatever is held by
  the time it finishes. Whichever selection finishes last decides what is
  kept; both spans are self-consistent, so the worst case is a later
  selection re-fetching.

Signed-off-by: F.N. Claessen <felix@seita.nl>

* ui/graphs: ignore the answer to a window no longer on screen

Context:
- Review feedback: the picker's selected handler starts its requests
  without aborting any still in flight, and whichever settles last draws
  itself and becomes previousResult. A slow earlier selection answering
  after a quick later one therefore put the older range on screen, while
  the picker and the URL named the newer one.
- The cache fix in 9928c43 addressed a different race, inside the cache.
  It kept what is held self-consistent, but said nothing about which
  answer the page draws.

Change:
- Count how often the shown window changes, claim that count when a
  selection starts, and drop the result if the count has moved on.
- Bump the same count when reloadChartData runs, since a reload for
  changed data supersedes any selection still in flight.

Signed-off-by: F.N. Claessen <felix@seita.nl>

* ui/graphs: guard the reload against being overtaken too

Context:
- Review feedback: 1b95e9c had reloadChartData bump the shown-window
  count so that a selection in flight could not overwrite it, but the
  reload never checked the count itself. A range selected while a reload
  was running could therefore render first, and the reload could then
  finish and draw the window the user had already moved away from.
- The guard was written in one direction only.

Change:
- Have the reload claim a count as the selection handler does, and drop
  its result if a window has been shown since.

Signed-off-by: F.N. Claessen <felix@seita.nl>

* ui/graphs: give the date picker the range it is meant to open on

Context:
- The initial load declared `var sessionStart` and `var sessionEnd` inside
  the DOMContentLoaded handler, far below the picker's creation. Being
  `var`, they were hoisted to the top of that handler and shadowed the
  module-level values assigned earlier.
- So createPicker(sessionStart, sessionEnd, ...) and, on simulation
  servers, computeSimulationRanges(...) both received `undefined` rather
  than the session's window. Confirmed by parsing the rendered page, which
  puts the declaration at line 341 and the use at line 217 within one
  handler spanning 170 to 434, and by running the same shape in a browser.

Change:
- Drop the two shadowing declarations and build the dates where they are
  used, so the module-level values reach the picker.

Signed-off-by: F.N. Claessen <felix@seita.nl>

* ui/graphs: guard the first load against being overtaken as well

Context:
- Review feedback: the picker is created before the initial load starts,
  and that load can take seconds on a wide window, so a range selected
  while it runs would render first and then be overwritten when the first
  load resolved.
- That is the third async path onto the chart, after the selection handler
  and the reload, and the only one still unguarded.

Change:
- Claim a count for the initial load and draw its result only if no window
  has been shown since.

Signed-off-by: F.N. Claessen <felix@seita.nl>

* docs: reference only the PR in the changelog entry

Context:
- Review feedback: the entry linked both the PR and the issue it closes.

Change:
- Drop the issue link. The other entries on this branch, and those on
  PR #2434 and PR #2435, already reference only their PR.

Signed-off-by: F.N. Claessen <felix@seita.nl>

* ui/graphs: drop the last comment claiming getAssetKPIs mutates its argument

Context:
- Found while adversarially reviewing PR #2434: this branch corrected the
  inline comment about the fastChartWindow snapshot but left the block
  comment above the declaration still giving the old reason, that
  getAssetKPIs bumps storeEndDate by a day. It does not, since bc6d5c1.

Change:
- Give the snapshot its real reason, matching the wording on PR #2434 so
  the two branches stay identical here.

Signed-off-by: F.N. Claessen <felix@seita.nl>

* ui/graphs: hand the KPI window fix to its own PR

Context:
- Review by @nhoening: PR #2434 now isolates the KPI window fix, so this PR
  should carry neither the changelog entry nor the code, nor its test.
- Almost right. What this PR genuinely needs is narrower than the fix: the
  advance must not reach storeEndDate or previousResult.end, because both
  are handed to the cache, which would then record a span a day wider than
  it holds. It does not need the advance gone.

Change:
- Restore getAssetKPIs to what main has, advance and all, so the fix and
  its user-visible effect belong wholly to PR #2434.
- Pass copies at both call sites, which changes no KPI value and keeps the
  chart's own dates out of reach. Once #2434 lands these are belt and
  braces, and the two branches merge either way.
- Drop the changelog entry and test_kpi_window_end_is_exclusive, which
  #2434 carries.
- Say why replay fetches around the cache rather than through it.

Signed-off-by: F.N. Claessen <felix@seita.nl>

* ui/graphs: stop reusing records indefinitely

Context:
- Review by @nhoening: the cache had no age limit. Data written elsewhere,
  by a report or another user, raises no event in this tab, so a window
  left open kept answering from what was true when it was opened. Changing
  the date range used to fetch, and so used to show such data; with the
  cache it no longer did.

Change:
- Hold records for five minutes. Long enough that stepping or widening a
  selection is still answered from memory, short enough that a page left
  open picks up data written elsewhere before long.
- Extending a span keeps its original age, so it cannot be kept alive
  indefinitely by being added to.
- The limit is a parameter of the cache, so tests can age it in
  milliseconds rather than waiting.

Signed-off-by: F.N. Claessen <felix@seita.nl>

---------

Signed-off-by: F.N. Claessen <felix@seita.nl>
Signed-off-by: Nicolas Höning <nicolas@seita.nl>
Co-authored-by: Nicolas Höning <nicolas@seita.nl>
@Flix6x Flix6x self-assigned this Sep 1, 2026
@Flix6x Flix6x added bug Something isn't working Data UI labels Sep 1, 2026
@Flix6x Flix6x added this to the 1.1.0 milestone Sep 1, 2026
Flix6x added a commit that referenced this pull request Sep 1, 2026
* ui/tests: run the JavaScript modules in a browser, from pytest

Context:
- The modules under ui/static/js had no tests, and the only way we had
  checked them was by hand. They are plain ES modules, so they can be
  exercised directly, but ES imports do not work over file://.
- A Node.js toolchain would be a heavy addition for ~5k lines of
  JavaScript, and would need its own CI job.

Change:
- Add a fixture that serves the modules over HTTP and runs a page of
  assertions in headless Chrome, reporting the results through pytest.
  Tests pass a snippet of JavaScript and assert on the checks it records.
- Allow a test to place the browser in a chosen timezone, so that
  behaviour around daylight saving is not tied to the machine.
- Add selenium to the test dependency group, which the runtime image does
  not install. The tests skip when selenium or Chrome is missing.

Signed-off-by: F.N. Claessen <felix@seita.nl>

* ui/tests: cover the date, data and chart-data helpers

Change:
- Test toIsoStringWithOffset, subtract and countDSTTransitions, the last
  under two fixed timezones, one with daylight saving and one without.
- Test getUniqueValues and convertToCSV.
- Test decompressChartData, including the pass-through of data already in
  the old format and the conversion of seconds-valued sensors to dates.

Signed-off-by: F.N. Claessen <felix@seita.nl>

* ui/static: fix two bugs the new JavaScript tests found

Context:
- toIsoStringWithOffset appended the local UTC offset to date.toISOString(),
  which is UTC, without moving the clock time. The result named an instant
  wrong by exactly that offset. Its only caller sends the KPI window to the
  API, so KPIs could cover the wrong days, by a whole day west of UTC.
- getUniqueValues ended its loop at the first falsy entry, and tested for
  values already seen with `in`, which consults the prototype chain, so a
  source named "constructor" or "toString" was silently dropped.

Change:
- Write the local clock time, then append the offset.
- Iterate by index, track what has been seen in a Set, and guard the nested
  lookup against a missing record.

Signed-off-by: F.N. Claessen <felix@seita.nl>

* docs: changelog entries for the JavaScript tests and the KPI time range fix

Context:
- One is infrastructure for plugin developers and hosts, the other is a
  user-visible fix.

Change:
- Add both to v1.1.0. PR numbers are placeholders until the PR exists.

Signed-off-by: F.N. Claessen <felix@seita.nl>

* docs: fill in the PR number in the changelog entries

Context:
- The entries were added before the PR existed, with XXXX placeholders.

Change:
- Point both at PR #2435.

Signed-off-by: F.N. Claessen <felix@seita.nl>

* ui/static: stop counting a record with no source as a source

Context:
- Review feedback on getUniqueValues: a record that lacks the key yields
  undefined, which then landed in the results.
- Its only caller, checkSourceMasking, counts distinct source ids to decide
  whether to warn that data is being masked, so one record without a source
  made it warn about masking that was not happening.
- This predates the null-entry fix in this PR: a record carrying no source
  at all already contributed an undefined.

Change:
- Skip values that are absent, and assert that a record without a source
  contributes nothing.

Signed-off-by: F.N. Claessen <felix@seita.nl>

* build: lock the selenium test dependency

Context:
- CI installs from uv.lock with --frozen, so adding selenium to the test
  group without relocking failed the pre-commit check and the image build.
- Locking with an older uv rewrote the file to an earlier revision, so this
  was done with 0.10.9, the version CI pins.

Change:
- Add selenium and its eight dependencies. Nothing else moved: no package
  was removed and no existing version changed.
- Apply black to the new test fixture.

Signed-off-by: F.N. Claessen <felix@seita.nl>

* ui/tests: format with the black version this repo pins

Context:
- CI failed on formatting the new test files. The repo pins black 26.3.1,
  which hugs multiline strings; the black in my environment was 25.1.0,
  which leaves them wrapped, so it reported the files as clean.

Change:
- Reformat with 26.3.1.

Signed-off-by: F.N. Claessen <felix@seita.nl>

* ui/tests: break the run() docstring only after punctuation

Context:
- Review feedback: the sentence wrapped after "daylight saving", against
  the repo convention, which PR #2433 has just extended to cover
  JavaScript as well as Python.

Change:
- Reflow so each line ends at a comma or a period.

Signed-off-by: F.N. Claessen <felix@seita.nl>

* ui/static: say in the JSDoc that the clock time written is the local one

Context:
- Adversarial review of PR #2434: the example did not say which clock the
  components come from, which is exactly what the fix changed.

Change:
- State it, and keep the file identical to the copy on PR #2434.

Signed-off-by: F.N. Claessen <felix@seita.nl>

---------

Signed-off-by: F.N. Claessen <felix@seita.nl>

@nhoening nhoening left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I believe the main fix is a good idea and I like one half of the tests that are being added. So the changes I ask for a overseeable.

However, I ran into a different problem while testing, which I believe might be good to fix here, as well - if we can identify the underlying problem.

My manual tests do not seem to get the right results yet, for multiple selected days.

The KPI is measuring energy costs, so it is summing up data (entry in sensors_to_show_as_kpis: {"title": "Daily costs", "sensor": 49173, "function": "sum"})

I had test data with ~122 EUR energy costs for Jan 15, 2030. I added 100 EUR for Jan 16, 2030 (note: different data source, as I uploaded one data point).

Selecting one day works well:

  • Jan 14: KPI widget shows 0 EUR
  • Jan 15: KPI widget shows 122 EUR
  • Jan 16: KPI widget shows 100 EUR

Here is a typical API query: http://localhost:5000/api/v3_0/assets/4497/chart_data?event_starts_after=2030-01-15T23:00:00.000Z&event_ends_before=2030-01-16T23:00:00.000Z&compress_json=true

Now I select multiple days:

  • Jan 14 + Jan 15: Total costs 122 EUR (GOOD)
  • Jan 15 + Jan 16: Total costs 100 EUR (FALSE, I expected 222 EUR ... API query)

In fact, all day ranges including Jan 16 give me 100 EUR...
Maybe this is an unrelated bug, as the behavior shows up on main, too (but with the original bug of selecting one day more). I also checked on #2435 as that has a fix for an offset issue, but observed the same outcome.

The values differ per day, so that a window covering the wrong days totals differently,
rather than merely covering the same number of days.
"""
from datetime import datetime

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This import and the next should go to the top of the module.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done — datetime, pytz.utc, GenericAssetType and TimedBelief are now imported at the top of the module, and the in-function imports are gone.

return asset, window_start


def _kpi_total(client, asset, start, end):

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This could use type hinting, certainly for the return value.
Same for the utility above.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done. _asset_with_daily_kpi returns tuple[GenericAsset, datetime] and _kpi_total takes (client, asset: GenericAsset, start: str, end: str) -> float.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not really convinced by these tests. They are testing some code artefacts, e.g. not using setDate() which has gone as of this PR, or using another function twice.

We should test outcomes if anything.

I do like the API tests added in this PR, and the JS code is basically wrapping an API call. IMO this test here can go.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agreed, and removed. You are right that asserting on setDate being absent tests how the page is written rather than what it does, and it would have gone stale on its own.

What it was guarding — that getAssetKPIs does not move the window it is given — is now covered by outcome instead: test_kpi_window_end_is_exclusive and test_kpi_window_honours_the_offset_it_is_given both fail if the window sent is wrong, and the new test_kpi_reports_what_the_chart_draws asserts the reported total equals what the chart draws. I checked each goes red for the right reason before keeping it.

nhoening and others added 6 commits September 1, 2026 11:52
Signed-off-by: Nicolas Höning <nicolas@seita.nl>
Kept both changelog entries: PR #2435 merged its own, this branch keeps its.

Signed-off-by: F.N. Claessen <felix@seita.nl>
Context:
- @nhoening found that multiple selected days gave the wrong total: with
  January 15 worth 122 from one source and January 16 worth 100 from an
  uploaded point, the two days together reported 100 rather than 222, and
  every range including the 16th reported 100.
- The cause is that the KPI read get_sensor_stats, which groups belief rows
  by data source, and then took whichever source the database returned
  first. Reproduced exactly, and on this machine the arbitrary winner was
  the other one, which is itself the point.
- The same reading has a second consequence: those stats aggregate every
  belief row, with no belief-time filter, so a revised value was added to
  the value it revised. One day believed first as 50 and later as 122
  reported 172, where the chart shows 122.

Change:
- Read the beliefs the chart draws, one value per event, and reduce those.
  A KPI is read beside the chart, so it should total what the chart totals.
- get_downsample_function_and_value now takes those values rather than
  per-source statistics. Its only caller is this endpoint; the sensor stats
  endpoint is untouched, since a per-source breakdown is the point there.

Signed-off-by: F.N. Claessen <felix@seita.nl>
Context:
- Review by @nhoening: the rendered-page assertions test artefacts rather
  than outcomes, the imports belong at the top of the module, and the
  helpers should carry type hints.

Change:
- Drop the rendered-page test. What it guarded, that getAssetKPIs does not
  move its window, is now covered by asserting the reported total.
- Add a test that a KPI totals exactly what the chart draws, over a sensor
  with two sources and a revised belief. It fails on the previous
  behaviour, reporting 100 or 172 where the chart shows 229.
- Hoist the imports, type the helpers, and correct the expectation of the
  offset test: now that the KPI reads the chart's beliefs, the two select
  the same events, so a window overlapping a fourth day counts it.

Signed-off-by: F.N. Claessen <felix@seita.nl>
…lexmeasures

Kept all three changelog entries, dropping the duplication the two orderings produced.

Signed-off-by: F.N. Claessen <felix@seita.nl>
Context:
- CI's flake8 caught F401: get_sensor_stats was still imported in assets.py
  after get_kpis stopped calling it. I had been skipping flake8 locally,
  which is exactly why this reached CI.

Change:
- Remove the import. The sensor stats endpoint still uses the function; the
  KPI endpoint no longer does.

Signed-off-by: F.N. Claessen <felix@seita.nl>
@Flix6x

Flix6x commented Sep 1, 2026

Copy link
Copy Markdown
Member Author

@nhoening — your three comments are addressed in the threads, and your bug is real. It turned out to be two, from one root. CI is green (13/13).

What you found

Reproduced exactly. On my machine the arbitrary winner was the other source — January 15 and 16 together reported 122 where you saw 100 — which is itself the diagnosis: there was no ordering at all.

get_kpis read get_sensor_stats, which groups belief rows by data source (GROUP BY DataSource.id), and then took whichever the database happened to return first:

downsample_value = dict(next(iter(sensor_stats.values())))["Sum over values"]

So with January 15 computed and January 16 uploaded — different sources, exactly as you described — only one source was ever counted, and every range including the 16th reported that one.

The second bug it uncovered

Those statistics carry no belief-time filter, so they aggregate every belief row. A revised value is therefore added to the value it revised. One day believed first as 50 and later as 122 reports 172, where the chart shows 122.

The underlying problem

Both come from the same thing: the KPI aggregated raw belief rows, when the chart draws one value per event. A KPI is read beside the chart, so it should total what the chart totals.

get_kpis now reads exactly the beliefs the chart draws (search_beliefs(..., most_recent_beliefs_only=True)) and reduces those. Your case gives 222, the revision case gives 122, and I verified KPI and chart agree on every window I tried.

get_downsample_function_and_value now takes those values rather than per-source statistics; its only caller is this endpoint. The sensor stats endpoint is untouched, since a per-source breakdown is the point there.

One consequence worth your judgement

The KPI now inherits the chart's window semantics. Previously it selected on event_start alone; search_beliefs selects events that overlap the window. For midnight-aligned daily sensors — what the endpoint documents — nothing changes. For a window that cuts across an event, the KPI now counts that event because the chart draws it. I think that is right, since the two should agree, but it does move numbers a third time in this PR, so I would rather you saw it than found it.

test_kpi_reports_what_the_chart_draws pins the invariant over a sensor with two sources and a revised belief; it reports 100 or 172 on the old behaviour and 229 now. I checked each new test goes red for the right reason before keeping it.

@nhoening nhoening left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I see new, but wrong results:

  • Jan 14: KPI widget shows 122 EUR (should be 0)
  • Jan 15: KPI widget shows 222 EUR (should be 122)
  • Jan 16: KPI widget shows 100 EUR (this seems good)

so the next day seems to get included (again)

Context:
- @nhoening found that after the previous commit every one-day selection
  also counted its neighbour: with data on the 15th and 16th, the 15th
  reported 222 and the 14th reported 122.
- Reading the chart's beliefs brought the chart's selection rule with it,
  which takes events *overlapping* the window. His sensor's daily events
  sit on the UTC grid while he reads them from +01:00, so every local day
  runs 23:00 to 23:00 UTC and straddles two events, and each event fell
  into two adjacent days.
- Reproduced only once the events were placed on the UTC grid; with events
  at local midnight the totals were right, which is what made the first
  attempt look correct.

Change:
- Keep reading the beliefs the chart draws, but count each event under the
  window it starts in. Summing what the chart draws would count a
  straddling event under both neighbours.
- This is a deliberate difference from the chart: the chart draws such an
  event in both days, the KPI counts it once.

Signed-off-by: F.N. Claessen <felix@seita.nl>
@Flix6x

Flix6x commented Sep 1, 2026

Copy link
Copy Markdown
Member Author

Reproduced and fixed in 6ddda9f — thank you, that was a regression I introduced in the previous round.

What went wrong

Reading the chart's beliefs brought the chart's selection rule with it. search_beliefs returns events that overlap the window, which is right for drawing and wrong for a total.

Your sensor's daily events sit on the UTC grid while you read them from +01:00, so every local day runs 23:00 → 23:00 UTC and straddles two events. Each event therefore fell into two adjacent local days and was counted under both. That is exactly the 122 / 222 / 100 you saw.

Worth saying: my first attempt to reproduce it passed. I had placed the events at local midnight, where nothing straddles anything. Only once I put them on the UTC grid did your numbers appear — which is also what pins down what is particular about your data.

The fix

The KPI still reads the beliefs the chart draws, but now counts each event under the window it starts in:

daily sensor on the UTC grid, read from +01:00
  Jan 15: 122    Jan 16: 100    Jan 17: 0      (was 122 / 222 / 100)

This is a deliberate, narrow difference from the chart: the chart draws a straddling event in both days, the KPI counts it once. So "the KPI totals exactly what the chart draws" — which I asserted last round — was the wrong invariant, and I have corrected that test to compare only the events a window owns.

Tests

  • test_kpi_counts_each_event_under_one_day_only covers your shape: UTC-grid daily events read from +01:00, asserting 122 / 100 / 0 and that the three days total 222 rather than 344. It goes red on the previous behaviour, with "the 16th must not also count the 15th".
  • The offset test is back to expecting 9.0, since selection is start-based again.
  • The multi-source and revised-belief tests are unchanged and still pass; that part of the fix stands.

flake8 and black clean, both at the versions pinned here. CI running.

@Flix6x

Flix6x commented Sep 1, 2026

Copy link
Copy Markdown
Member Author

@nhoening Actually, what is your expectation to see as KPIs if you have daily sensor data in UTC and you view from, say, Tokyo?

@nhoening

nhoening commented Sep 1, 2026

Copy link
Copy Markdown
Member

Thanks, I can confirm the behavior makes sense now!

@nhoening Actually, what is your expectation to see as KPIs if you have daily sensor data in UTC and you view from, say, Tokyo?

Good question. You mean if the KPI data has no offset? I guess I would like a translation to my local timezone. But this isn't easy to adjust and/or to explain. I would like KPIs to be calculated on the same offset as input sensors (mostly the power sensors that matter), in the first place. But of course we cannot assume that. Or should reporting make that possible so KPIs make sense?

It seems to me this is well worth of its own PR? Maybe a Claude design session first, to create an issue?

@Flix6x
Flix6x merged commit f1911a9 into main Sep 1, 2026
13 checks passed
@Flix6x
Flix6x deleted the fix/kpi-window-off-by-one branch September 1, 2026 17:23
@Flix6x Flix6x modified the milestones: 1.1.0, 1.0.1 Sep 9, 2026
Flix6x added a commit that referenced this pull request Sep 9, 2026
The v1.0.1 patch release backported #2434, #2461, #2465, #2473, #2475
(partial - only the ad98460751d9 migration fix), #2479, #2480, #2482,
#2499, #2502 and #2507 from this history to the 1.0.x branch. Move their
changelog bullets out of the v1.1.0 section into a new v1.0.1 section, so
the same fix is not listed as unreleased once 1.0.1 ships; the CLI
changelog gets a matching 'since v1.0.1' section. The API change log
keeps its own independent v3.0-NN numbering and is untouched.

Signed-off-by: F.N. Claessen <claessen@seita.nl>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working Data UI

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Asset KPIs count one day too much

3 participants